Interactive Media Display Across Devices

ABSTRACT

A computer-implemented method includes identifying a computer-based portable program module, automatically altering code in the portable program module to permit display of the module on a television-based display so that the displayed module has a substantially similar appearance on the television-based display as on a computer display, and providing the altered code for execution on a processor connected to a television-based display.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to U.S. Application Ser. No.60/909,384, filed on Mar. 30, 2007, the contents of which are herebyincorporated by reference in its entirety.

TECHNICAL FIELD

This document discusses systems and methods for displaying and sharingmedia content and other objects for television viewing.

BACKGROUND

As the PC and the television continue to converge, both in deliverymodels and interaction models, we face a number of new challenges andopportunities. These devices, communicating through a network in aloosely coupled manner, will serve different and overlapping roles inpresenting entertainment options. Commonly, the PC is used by theindividual, used for many non-entertainment tasks and often located awayfrom the main entertainment room. In contrast, the television is in thecenter of the entertainment space, often used by groups of people invarious social contexts, has an impoverished input device (the remote)and has a lower resolution display. It lacks computational power, uses adifferent model for content distribution, and a different model formedia presentation (channel-based).

Today we are seeing the beginnings of a federated model, where the PC isused to store content and the TV (or set top box) uses the localinternet system to access and present the content. As computers becomeless expensive, some of this will move directly onto the TV—small harddrives and internet access will make TVs capable of getting at webcontent without requiring a PC. Nonetheless, some of the interaction islikely to remain on the PC. You may receive recommendations for video,gather content links and share links while using a PC. The TV and the PCwill work in a loosely coupled fashion to support theCollect-Relate-Create-Donate (CRCD) model.

SUMMARY

This document describes various features and techniques for sharingdigital content and for providing and modifying portable program modulessuch as widgets or gadgets. In one implementation, acomputer-implemented method comprises identifying a computer-basedportable program module, altering code in the portable program module topermit display of the module on a television-based display so that thedisplayed module has a substantially similar appearance on thetelevision-based display as on a computer display, and providing thealtered code for execution on a processor connected to atelevision-based display. The computer-based portable program module maycomprise a gadget, and the code in the portable program module maycomprise mark-up code. Altering the code can comprise altering stylesheet definitions for the module. The method may also include alteringgraphical elements referenced by the portable program module to bedimensioned for the television-based display.

In some aspects, providing the altered code for execution comprisestransmitting the code in response to a request from a remote set-topbox. In addition, altering the code in the portable program module maycomprise generating code responsive to navigation commands from atelevision remote control keypad. Generating code responsive tonavigation commands can also include generating four-direction code fornavigating between input elements generated by the portable programmodule. The method may also comprise transmitting data in response to arequest from an executed portable program module, and can also compriseencoding the data as XML data.

In other aspects, the method also includes automatically altering thecode in the portable program module, and requesting confirmation from auser for one or more alterations. The method can also include displayingan executed module on a computer monitor as part of a simulatedtelevision display.

In another implementation, a method of displaying content on a videomonitor is disclosed, and comprises receiving, from a viewer of atelevision, a command for display of computer content on the television,generating a display from a portable program module with a display ofmedia content on the television, requesting over the internet, byexecution of the portable program module, data for display with theportable program module. The command may be received from a televisionremote control. The portable program module can comprise a gadget. Also,the portable program module can comprise mark-up code, includingJavaScript referencing code. The portable program module can alsocomprise mark-up code for navigation of input controls on a displayedmodule using four-direction navigation.

In other aspects, the method further comprises positioning the displayfrom the portable program module in a pre-selected corner of atelevision screen. The method may also comprise generating a pluralityof displays from one or more portable program modules in an array, andthe array may be a linear array. The linear array may be generated, forexample, along a side edge of the television screen. The method can alsoinclude changing the display of media content from a 16:9 aspect ratioto a 4:3 aspect ratio, such as when portable program modules aredisplayed along one edge of the display.

In another implementation, a method of sharing media content isdisclosed. The method includes receiving, from a viewer of a television,a command for display of a friend list, displaying indicators for aplurality of friends associated with a user, and receiving a selectionof one or more friends from he plurality of friends, and causing anindicator of a media program playing on the television to be sent toaccounts associated with the one or more friends. The indicators for aplurality of friends may each comprise a name of a friend and an imageassociated with the friend. The method may further comprise contacting asocial networking site to obtain information identifying the pluralityof friends. Also, the indicator of a media program comprises a URLpointing to the a file of the program.

In yet another implementation, a method of obtaining media content isdisclosed. The method includes receiving, from a viewer of a television,a command for display of recommended media content, identifying one ormore friends associated with a user of a system and displayingrepresentations of content recommended by the one or more friends, andcausing media content recommended by the one or more friends to bedisplayed on the television. The method may further comprise displayingindicators for a plurality of friends associated with a user andreceiving a selection of one or more friends from the plurality offriends, and displaying representations of content recommended only bythe selected friends. The representations of content recommended by theone or more friends can also be displayed in one or more shelves.

In some aspects, the representations of content recommended by one ormore friends are displayed in a single shelf, and other recommendationare displayed in one or more other shelves. Also, causing the mediacontent to be displayed can include causing bumper content to bedisplayed between adjacent pieces of media content, and the bumpercontent can comprise previews of following media files. The method mayfurther comprise receiving a skip input from a user while bumper contentis being displayed, and in response, causing a media file associatedwith the bumper content to be skipped. Also, the bumper content maycomprise advertisements. In other aspects, causing media content to bedisplayed can comprise accessing streaming media over the internet andproviding the streaming media for display on the television.

The details of one or more embodiments are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description and drawings, and fromthe claims.

DESCRIPTION OF DRAWINGS

FIG. 1A shows an example television display of shelves of media contentorganized by the source that recommended the content.

FIG. 1B shows a screen shot of shelves o recommended content for a view,with one particular file selected.

FIG. 1C is a screen shot of an example of a bumper showing programdetails.

FIG. 1D is a screen shot of a friends gallery showing images associatedwith a plurality of friends.

FIG. 1E is a screen shot of a details page for a selected friend.

FIG. 1F is a screen shot of an interface for selecting a friend orfriends with which to share a media program while watching the program.

FIG. 2A shows a graphical flow diagram 200 of the gather of programlinks in a social media sharing application.

FIG. 2B. shows a video gathering application that displays friends towhich a user may drag representations of media files for association ofthe files with the selected friends.

FIG. 2C shows detail information for a friend in the application of FIG.2B.

FIG. 3A is a schematic diagram showing the recommendation on a personalcomputer of a video file, and the subsequent delivery of the file to atelevision of the recommender or of a friend of the recommender.

FIG. 3B is a schematic diagram of a system for sharing audio-visualprogram content.

FIG. 3C is a block diagram of a media display management system.

FIG. 4 shows a schematic diagram of a system for sharing and displayingmedia content.

FIG. 5 shows an example of four-direction navigation controls for atelevision remote control.

FIG. 6 graphically displays organization of media programs or shows, andintervening bumpers.

FIG. 7 is a screen shot showing a TV widget superimpose on a videodisplay.

FIG. 8A is a flow chart of a process for converting a portable programmodule, such as a gadget, from a computer format to a television format.

FIG. 8B is a flow chart of a process for sharing audio-visual contentamong friends and/or devices.

FIG. 8C is a flow chart of a process for sharing audio-visual contentamong friends.

FIG. 8D is a swim lane diagram of a process for sharing audio-visualprograms among devices.

FIG. 9 shows an example of a generic computer device and a genericmobile computer device.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

This document addresses, among other things, three problems related tothe convergence and contentions between the PC and the TV. The firstproblem relates to finding good content from the enormous amount ofcontent that has suddenly become available online. The second problemrelates to the delivery of online content to the entertainmentenvironment, and the third problem relates to managing this content(viewing it, sharing it, and opening doors for othercon-tent/interactions while watching television).

Problem—Finding Good Content:

YOUTUBE and other such services are filled with a vast and rapidlygrowing library of user-generated, commercial-generated, informal, andprofessional-level video content. The content not only spans a broadrange of topics, but also a broad range of aesthetic, visual, andsemantic quality. This large, ever-changing and varying set of topicsmakes finding “good” videos a challenge. The “what's a good video about”discovery problem is further exacerbated by the difficulty people havein expressing what is interesting to them.

To address this problem, researchers are working on systems toautomatically extract features and content tags that can be used to aidusers in searching through video content. Outside of content analysis,social networks are proving to be a powerful source of information forsifting through the vast amount of content. Not only are millions ofpeople a powerful parallel filtering system, but they can provide aninvaluable source of tagging meta-data, clustering, recommendations, andpopularity rankings.

One of the common mechanisms through which users find content is throughemail. With YOUTUBE's immense popularity and volume of content, peopletypically share content by emailing each other links. This points to theneed to support easy sharing on the PC, and allows us to examine whetheremail is the best option. Email is familiar, general, great fortransport and readily available. But, for a task-specific goal ofsharing links with a small set of people, email may be more complicatedthan necessary. Email requires several steps to compose, target, anddeploy the links, as well as several steps and separate tools to extractthe link, resolve it to video, and view it. Because email is used for avariety of information sharing tasks, people must sift through the emailor jump from one task to the next as they process email messages. Oftenpeople view videos individually and this lengthy viewing interrupts theflow of other email-related tasks. Because of this interruption, usersmay be less inclined to explore other videos and content, and manyinstead return to the task of reducing their email queue.

Problem: Delivering Content:

Currently, the prevalent mode for watching internet content is streamingover the internet onto a PC. This model provides a sub-optimal viewingexperience and limited range of content because of necessarybandwidth-to-latency trade-offs. We currently cannot experiencehigh-quality, feature length films delivered in real time. Instead, weusually click on content, wait some period of time for a partialdownload, only to watch low fidelity content that is necessarily shortand often temporally discontinuous (hitchy). For the viewer, thisdestroys any sense of immersion. For the content creator, this restrictsthe content that that they are willing to produce, the poor quality ofservice for streaming video beget lower quality, shorter video, andlittle incentive to create more than quick clips. In contrast,independent film industries have repeatedly demonstrated that there is alarge, untapped reservoir of longer playing, high quality (LPHQ) contentready for consumption, but little means for unhindered mass.

Problem: Managing Content on a TV:

A third problem revolves around where and when the user may want to viewthe content. Email typically delivers the link to a personal computingenvironment rather than to a social entertainment environment such asthe living room. Sifting through email, extracting the video andwatching it as part of an email-processing task is a very differentexperience than being in an entertain-me mindset, turning on a TV,watching it with a group of friends and using a remote control tonavigate through the content. However, naively browsing on a TV will notwork either. If the sea of internet-based, user-generated content issuddenly available on a TV (via a set top box), we will need newinteraction models that deal with the amount of content and the desireto connect with the communities who share the content. This is a greatopportunity for the User Interface/Experience community to pioneerbeyond the pick-and-play models of PVRs today.

To address some or all of these problems, we discuss an example systemthat integrates the PC and the television via the internet. The hardwarefor the TV, in this example, is a set top box prototype that developerscan construct from a low-end laptop and remote control. The examplesystem is designed to gather content and play it though an SVGA cable toan ordinary television—though other connections such as HDMI and DVI mayalso be employed. This means developers can easily purchase one andstart developing and extending the system without any special hardware.This also gives researchers a concrete system for developing and testingthe complete experience. Other hardware solutions may also be used toperform similar functionality, including standard set top boxes andelectronics integrated with a television itself.

FIG. 1A shows an example television display 100 of shelves of mediacontent organized by the source that recommended the content, using theexample implementation, while FIG. 1B shows a more direct view of thedisplay 100. In this example, a graphical user interface is displayedshowing audio-visual programs that have been associated with friends, oracquaintances, of the owner of the television, where the television hasbeen logged onto an account of the television owner. The organization ofthe content is in horizontal rows or shelves, where each programincludes a still frame of video from the associated program, and canalso include an image or other indicator of the friend from which thecontent was provided. Certain content may have been provided by thetelevision owner himself, such as videos identified by the owner when hewas previously using his desktop computer.

The system provides a familiar remote control interface. A user turnstheir device on, they see shelves of content gathered from today's (or alonger recent time) recommendations, and can easily skip or startplaying along a shelf. Recommendations come from friends (their socialnetwork), from content analysis, and from persistent searches. Inparticular, a first shelf 102 shows programs from friends, as ahorizontal row of still frames from the relevant content (audio contentmay show meta data about the audio file such as title, artist, andlength) having a superimposed image of the friend that shared thecontent. The content on shelf 102 may be sorted or filtered in a varietyof manners, such as by friend, where the top friends may be friends thatthe user has identified as people who share good content, or bycomparing ratings that the user and the friends have given to commonpieces of content and making a determination of co-variance in tastesbetween users (where higher correlation makes a friend's recommendationsappear higher in the list of recommended programs).

Shelf 104 shows programs that the user himself provided. As discussedelsewhere, the user may have dragged certain content onto their devicewhile they were surfing the web on their desktop computer, and suchcontent would show up on shelf 104. One particular program—Ryan vs.Dorkman2—is shown in an enlarged, highlighted state with a lit border108 in the middle of the screen 110. In this example, the user hasnavigated to that program through the two-dimensional grid of programsusing a 4-direction controller of a television remote control.

Shelf 106 shows other recommendations, such as recommendations madeautomatically by the system. Such recommendations may be made, forexample, by comparing a user's interests (as indicated by programs theuser watches, or that the user watches without skipping out of, or thatthe user rates highly) to those of others, and providing recommendationsof programs that similar users liked. In a similar manner, a user mayestablish persistent searches for content, and the recommendations mayincluded the results of such persistent searches. For example, the usermay provide a key word like “funny cat stunts,” and may then be providedwith all new videos that show such stunts. The persistent search maykeep track of content the user has previously viewed or declined toview, or may only show content from a most recent time period, so thatthe user is presented only with fresh content.

The particular display shown in FIG. 1B is at a middle zoom level fordisplaying content on a television. If the user hits “back” on theirremote control, they can be shown smaller view of the shelves. Such a“globally” zoomed view clusters shelves together. It relies on animationand pre-biasing to help recognition even when the images are very small.When the user selects a particular piece of content, the system zooms inon that piece of content to the highest zoom level of detail as shown inFIG. 1C.

FIG. 1C is a screen shot of an example of a bumper 111 showing programdetails. The bumper 111 can be reviewed by a user to see program detail,and can also be shown for a short period of time immediately before thecorresponding program begins to play (e.g., when programs, such asadjacent programs on a single shelf from FIG. 1B, are being playedautomatically in succession). The bumper gives the viewer a chance topause or jump from bumper to bumper in a linear fashion. Hitting theback button while on a bumper zooms the viewer back out to the shelfoverview.

Bumpers, whether of program information or advertisements or otherinformation, may also be shown simultaneously with a program, such as inthe form of a “bug”. A bug is a visible item that takes up less than asubstantial portion of the display, and may provide a wide variety ofinformation to a viewer. The big, in certain circumstances, may becaused to generate at various times during the playing of a program soas to act like an in-line commercial, and can be particularly generatedas part of a pop up programmable program module of the forms discussedbelow.

The bumper 111 includes a title area 112 that shows the program titlealong with a short description of the program. An image area 114 shows astill image from the program, though a bumper 111 for a program like anaudio program may simply show a logo or an album cover. Detail area 116shows particular relevant details about a program, such as the programlength and a composite rating level the program has received from otherusers, a URL for the program or a web site associated with the program,and tags that have been attached to the program. When the bumper isdisplayed as part of a queue of programs being displayed in sequence, itmay be displayed for a pre-determined period and then rolled up so thatthe corresponding program may be played.

A comment area 118 shows comments on the program from other users suchas friends of the first user. The comments may be accompaniedautomatically by images of the commenting user, in a familiar manner, topermit faster navigation of the comments (e.g., so that a user can morequickly locate comments from someone they respect). Controls may also beprovided, such as for the first user to leave comments, or for the firstuser to complete a message to one of their friends, such as a noteplayfully castigating a friend for recommending a program that the firstuser thought was horrible.

While interacting with the display of FIG. 1B, a user, much like withvarious digital video recorders (DVRs), can “trick play” the content:they can pause, rewind, fast-forward, or skip to the next item orprogram on the shelf. If they do nothing, once a video starts playing,the system will play the next item on the shelf. In between each show,or as a briefly generated overlay bug, an automatically generated“bumper” briefly describes the next show (including who recommended it,annotations, etc.). A video buffer showing a commercial may also bedisplayed between programs.

In this example, the highlighted selection remains in the center and thecontent animates underneath of it (like a magnifying lens) for aselected program. Each successive selection of “OK” or “select” on theremote selects the program, brings up more details, and then beginsplaying the program. This causes a smooth transition from the spatialmetaphor of the shelf to the temporal model of a channel. Unlikepick-and-play models in current DVRs, when this content is done, it thenproceeds to the next piece of content on the shelf. Bumpers between thevideo allow a glimpse of the spatial model if the user wants to smoothlytransition back to the shelves, and also can show advertisements orother automatically selected content between programs that are on theshelf.

The model of using a bookshelf to represent rich media is familiar(Card, S. et al., supra and Kukuchi, H., “User Interface for a DigitalLibrary to Support Construction of a “Virtual Personal Library.” 1996Int'l Conf. on Multimedia Computing and Systems (ICMCS'96) (1996)). Morerecently, APPLE ITUNES on the PC and APPLE TV on television haveprovided navigation metaphors.

There are several advantages to using a bookshelf metaphor. It allows auser to place various, heterogeneous objects on it (music, video, etc.),it is a familiar mimetic, and it has a close correlation or mapping toplay lists and ranked search. Order is implicit from left to right inthe design and hitting play starts at the given item and continues tothe right, looping when the end has been reached. In addition, loopingmay occur on the same shelf (e.g., repeat) or from one shelf to thenext. Bookshelves can also represent other groupings, for example apersistent search query such as “surfing,” and gather all availablesurfing videos each day.

Bookshelves can be stacked and stacks can be hierarchical. This allowsfor scaling to large numbers of items. Visually, the system can pre-biasthe viewer so that even at small sizes they will still recognize items.Shelves can also be sorted and filtered (e.g. by time, person, ranking).And recommendations/related videos can be highlighted on the shelf (oranother shelf). The ability to define shelves explicitly and implicitly(search, recommendation) and then sort them is a simple yet powerfultool. Shelves also allow people to create and share at a level above theindividual content. For example, a user can “subscribe” to the JonStewart shelf and see what videos he recommends (presumablyhigh-intelligence comedy work).

While watching or listening to a program, if the user decides to sharethe content, pressing a button on their remote control can bring up alist of friends 140, as shown in FIG. 1F, and the user can share thecurrent content easily with them. The list of friends 140 may beobtained, for example, from a list stored on the user's local clientdevice, but synchronized with a central server. The list 140 may bespecific to the sharing application, or may be a more general list ofrelationships that can be accessed from a social networking system,through which the user manages many other friend-of-friend activities.The playing program may be paused automatically until the sharingactions are complete.

In the example display of FIG. 1F, a display for sharing content withfriends is shown, where the list of friends is shown simply as avertical list of names and corresponding images of the friends, laidover the main television display. Such a display may be generated at anypoint where a user indicates an intent to share content that they arereviewing. The currently selected friend here is Betty McDonald 148, andpressing by the user of up and down buttons on their remote will causethe list to scroll so that the selection bar appears over others in thelist, such as Sara Silverstone 146. Other similar UI mechanisms may beemployed to permit easy selection of friends or groups with whom theuser would like to share content.

When a sharing selection is made in this example, the user may be askedto provide a comment on the program. Alternatively, if the user does notwant to type using a remote control, their account may indicate a needto enter a comment, and they may be asked to enter the comment the nexttime they log into the system on a computer. The sharing may thus beheld in wait until the person provides such comments from theircomputer. The comments may also be provided via a different mobiledevice that is identified in the user's account as being associated withthe user, such as a smartphone. Thus, for example, when the user makes aselection to share with a friend using their remote control (or using IRfunctionality on their smartphone as a form of universal remotecontrol), a message may be sent to their smartphone, such as a textmessage soliciting a comment on the program. The user may then respondto the text message by providing a comment, and the returned textmessage from the smartphone may be joined with the recommendationreceived via the user's television device, and may be provided to thefriends. In this manner, the user may take advantage of the relativestrengths of the general devices that they already own—i.e., televisionuser interface navigation with a remote control, communication abouttelevision programming with a set-top box or similar mechanism, andentering alphanumeric information with a computer or wirelesscommunication device.

Thus, using the system described here, a user may identify programs thatlook interesting as they surf the web, and may send such programs totheir home television. When they get home, they can watch the programs,and if the programs are good, the users may easily distributed theprograms to their friends along with comments. In a similar manner, theycan watch or listen to programs distributed by their friends, andforward the programs on to others of their friends if they enjoy theprograms. They may also schedule the programs into blocks ofprogramming, or may watch programming blocks arranged by their friendsor by perceived authorities in organizing enjoyable programming. Assuch, the disclosed systems and techniques permit simple socialtelevision programming (SSTP).

Since playing is a level of zoom, this model simplifies the transitionfrom organizational overview (shelves) to actually playing of thecontent. This model of playing is a hybrid of the temporal channel modelof traditional television and the “pick-and-play” model of most PVRs.Allowing automatic playing means that the shows can run in theback-ground while your main attention is elsewhere, but you can alwaysskip back or find stuff on the shelf, relieving the anxiety aboutmissing something. During playing, hitting the dedicated buttons orpause allow a viewer to easily recommend the content to their friendswithout the intervention of a desktop computer.

As shown in FIG. 1D, a similar shelf model can be used for accessingfriends. People can be sorted by frequency of contact (sharing ormessaging) or alphabetically, among other things. The details about aperson include the list of video that they have shared with the user andthat the user has shared with them. Since they see the same thing, thiscan be a useful place to have media-mediated conversations, see if theyare online, etc.

In FIG. 1D, a friends display 120 includes a search area 122, where auser may enter terms to search for a particular friend that is in theirnetwork. A particular friend 124, here Tina Lindsey, has been navigatedto by the user, with directional keys on a remote control, and is shownhere at a higher zoom level. The higher zoom may occur automaticallywherever the cursor is located.

If the user makes a selection on friend 124, then the additional detailmentioned above may be displayed in a pop-up box, as shown in moredetail in FIG. 1E. As shown in FIG. 1E, a display 130 shows a meta dataarea 132 that shows a search box (for searching across the user'scontent), an e-mail address of the friend 124, and other data about thefriend. A social area 134 contains selectable controls to permitcommunication with the friend 124 by various simple mechanisms such aschat or e-mail. An incoming area 136 shows content that the friend 124has shared with the instant user, while an outgoing area 138 showscontent the instant user has shared with the friend 124. These area maypermit the instant user to determine the sorts of programming that thefriend likes, and to thereby match his or her recommendations to thefriend to such other programming.

Referring again to FIG. 1D, certain users may be highlighted on thedisplay, such as by a lighted dot 126 or other visual mechanism (e.g., acolor border around their image or a pulsing of their image). Suchmechanisms may indicate, for example, that that friend is currently online so that chat communication may occur, or that the friend is a“power user” from whom the instant user has received many very goodrecommendations. Also, friends who are on line simultaneously may decideto watch a program simultaneously, so that, for example, they cancomment back and forth on the program, either via chat or audibly (e.g.,using microphones and headsets).

User Flow Example

An example implementation includes a PC-client application dedicated tomaking link sharing among friends simple and light weight (thoughsimilar functionality can be achieved using a set-top box, or may beintegrated directly into a television display, particularly if thedisplay supports a native web browser). The PC-client also supports morecomplicated tasks such as viewing and organizing the content, but not atthe cost of sharing fluidity. The system has a web repository to store,manage, and access the links and a TV-client system, in this example.The three-system model supports background downloading so thatlong-playing, high quality content is available in the entertainmentdomain where a user may watch several shows. The user gathersrecommendations (and the user's friends may also do the same) throughoutthe day, and the system can automatically fetch and prepare them forviewing at home, including by transmitting the content at a period oflower network load.

FIG. 2A shows a graphical flow diagram 200 of the gathering of programlinks in a social media sharing application. This illustrative exampleof the use of a media sharing system may show how a user can interactwith his computer to share media program files with friends, groups,and/or devices. The figure shows two web pages 204, 206, and a gatheringapplication 202 that may be displayed as a floating tool over a user'scomputer desktop, or may be part of a browser toolbar (not shown). Thegathering application 202 acts like a bucket for collecting URL links orother such indicators associated with audio-visual programs, such asvideo files, audio files, podcasts, digital image slide shows, and thelike. Where the user drags a link to the gathering application 202 andthe link is accessible to the other users, that link may simply be madeavailable to the other users. Where the first user drags a file to thegathering application 202 and the file is not otherwise available (e.g.,it is a video of the user's baby stored on the user's local drive), thefile may be copied to a central location where it can be accessed by theother users, and then a link to the file may be provided to the otherusers (and the file automatically downloaded to the other users' devicesin certain implementations). The file may be deleted after the last userhas reviewed it.

The gathering application 202 includes a number of graphical receivingsections that represent persons, groups, or things to which content maybe deposited. For example, user 208 may be a friend of the person who isrunning the application, and dragging of content to their area on thegathering application 202 may cause the content to be made available tothat person conveniently, as described in more detail above and below(e.g., FIG. 1B).

In addition to sharing content with other users, content may be sharedwith a device 212, which may be a particular audio-visual device, suchas a set-top box or other television peripheral. The user of thegathering application 202 may use such an interface to have contentdeposited on his device at home, or onto his mobile smartphone. The userof the gathering application 202 may instead identify himself as a user210 and may drag content to himself in appropriate circumstances.

There can be a difference in the action of dragging content to a device212 versus dragging it to a user 210, however. As described below, usersmay simply be presented with content as an option when they get home andturn on their television, without the content having been previouslydownloaded to their device. That is because, if they have a large numberof friends, the collected amount of media they might receive could besubstantial, and could consume unnecessary network bandwidth. Thus, insuch a situation, when sharing with a friend occurs, only links to thecontent may be created in a profile associated with the user who ishaving the content shown to them, and that profile may be accessed froma central system when the user sits down in front of their television.

In contrast, actions of sharing with a device may cause the actualcontent to be downloaded to the device in the backgroundautomatically—such as while the user is still at work. Such additionalbandwidth usage may be tolerable because the user, having direct accessto the device, can be presumed to be the device owner or someone in arelatively small group that has been given direct access to the device.Also, it can be assumed that the recipient of the content really wantsto see the content, and thus would have burdened the network with itsdownload when they returned home, in any event. The actual downloadingmay be immediate or nearly immediate, or may be scheduled. For example,a user may establish an account whereby they indicate that they do notwatch television at home until after 6 p.m. on a weekday. If the userdrags content to their device while they are on their lunch break atwork, the download of the content to their home television may bedelayed until late afternoon because networks may be abnormally busyover the lunch hour, and the downloading of video at that time may placean unnecessary strain on the network. The user may also be given theoption, at the time of dragging material to the gathering application202, to make the download immediate or scheduled, e.g., such as havingtwo zones on the image of the device or by having a right-click-and-dragoperation by which a contextual menu permits a choice of “immediate” or“scheduled.”

Group identifier 211 provides sharing that is similar to that for theuser 210, but across a defined group of users. For example, a user mayestablish a program distribution list, much like establishing an emaildistribution list, of a group of friends with whom they like to shareaudio-visual content. A single dragging action may then sendnotifications to the accounts of all those users. A group may alsoinclude other groups. A person or group can be in multiple groups, sogroups are not strictly hierarchical (and cycle detection may occur toresolve the final set of people with whom to share). From a developerAPI's standpoint, a group can have a unique ID and can be identifiedbased on the creator of the group (a person ID) and the name of thegroup (group names are unique within a person who creates them). In thismanner, developers may provide extensions to the disclosed system bywhich to provide for even further social interactions between users andtheir media.

The group may also be an “opt in” group, whereby the first user sets upthe container of a group and other people can join the group as theylike. In this manner, a user my make themselves an authority in aparticular area and may, in effect, become a program manager. Otherusers who like the style of the first user may then subscribe to thefirst user's “channel.” As one example, the first user may be a DJ whois very good at finding breaking music acts, and may thus put together aplaylist by dragging links (e.g., off of band pages on a socialnetworking site) to the gathering application 202 (and may subsequentlyuse an application to re-order the content) and make it available to thepublic. When other users who have subscribed to the first user's“channel” get home, they can turn on their home entertainment system,and have the programmed playlist played while they prepare dinner. Thefirst user may also apply post-selection alterations such as addingvoice commentary between programs, such as songs. Thus, for example, aclassical music expert could provide brief comments before each songthat they play on their channel. Such user-programmed media systems arealso described in co-pending U.S. patent application Ser. No.11/742,495, filed Apr. 30, 2007, which is incorporated herein byreference in its entirety, and particularly with respect to itsdiscussion of custom media channels that can be formed by users of asystem.

Where a user wishes to drag content to the sharing application 202 butthe friend or group they would like to share with does not appear there,they can drop it into a “temporary” drop 207, shown in FIGS. 2B and 2C.The temporary drop holds an indicator for the content, such as a URL,but does not send it to any one else's account or to a device. Rather,the temporary drop 207 simply holds the content indicator until the useris able to search for, locate, and add the friend or group with whichthey originally intended to share the content.

In one particular example of the operation of such a system, during theday, a user may receive an email from their friend Bob with a YOUTUBElink in it (not shown). Instead of resolving the link and watching itnow, the user can simply drag it to the gathering application 202. Theuser can also drag the link to other users 210 in the list to quicklyshare with them or with groups. This allows the first user to extract,very easily, recommended links from external systems and import theminto the sharing system 200.

This can also create a chain or graph of people through which thecontent is shared or flows. Such information about the graph throughwhich the flow occurs, and the timing of the flows can be used toconduct flow and graph analysis. For example, it can be determined thatsomething suddenly gets shared a lot, or that everything shared fromRich to Bob, was then shared by Bob to Fred, making Fred a “hidden”friend of Rich. Similar sharing patterns (temporal and structuralsimilarities) can be used to recommend content from other people withoutrevealing those people to each other. Also, “hot” content that is beingshared rapidly can be identified, even within small groups of people(long tail). And “hot” people whose sharing is indicative of a largerdemographic can be identified (e.g., everything Bob shares, geeks love).Such a graph can be referenced as a POG or People-Object-Graph.

If a user clicks on his or her identifier in the gathering application202, he or she can be shown a list of programs that are currently queuedfor them (FIG. 2C). The programs may be shown in a list in the orderthey are currently queued to be shown, along with a title of eachprogram, a length in time of the program, and the person who shared theprogram. In addition, though not shown, an icon or frame around theimage (e.g. a film strip around the picture of the video a picture framearound an image, a curl of paper around a web page) may be provided toindicate the type of each program, such as short video (e.g., clip),long video (e.g., movie or television program), podcast, audio file,etc., and whether the content is free or requires payment. In addition,a search box is provided so that the user may find particular programsif the list of queued programs is very long.

The display of FIG. 2C also shows general social features, such as anindicator of how many friends of the user are currently on line. Chatfeatures (not shown) may also be provided, and the gathering application202 may be integrated with a chat toolbar, such as by augmenting atraditional chat toolbar with the ability to drag content to friends inthe toolbar. The user may also direct-drag items that others have sharedwith them, out of their list, and onto others of their friends (thoughthe system may prevent dragging to the friend who shared it in the firstplace).

Referring again to the example associated with FIG. 2A, Bob may havesent the first user a recommended link and the first user may thenimmediately share it with others in his or her friend list that are notin Bob's list. No other applications or tools would be needed, in thisexample, to share this link, such as an e-mail application or other suchapplication.

When the user goes home, their set top box can have been diligentlymonitoring their link repository (stored, e.g., on a central server thatcommunicates with the gathering application 202) and pre-fetching thecontent, or the repository may have pushed the content out. Since ithappens as a background process, the user can download longer content ata higher quality and have it ready for playing without hitching. (Thecontent can be encrypted to protect rights and the encryption itself canbe updated or changed over time.)

As described further below, “bumpers” may be placed between mediaprograms, such as commercials and other similar promotional items, asshown by schedule 600, program 602, and bumper 604 in FIG. 6. Thecommercials may be targeted and matched to the programs in familiarfashion such as by using tags associated with audio or video files askey words for ad selection. Also, where a first user produces a programline up for a plurality of other users, the first user may be provided,by a system managing the sharing, with a share of the ad revenue.Alternatively, the user may be given the opportunity to insert one ormore of his or her own bumpers from time to time, so that he or she maysign up his or her own sponsors and obtain remuneration for his or herprogramming efforts.

FIG. 3A is a schematic diagram showing the recommendation on a personalcomputer of a video file, and the subsequent delivery of the file to atelevision of the recommender or of a friend of the recommender. Thereare three parts to the exemplary system 300: (1) a computer client 306working with a gathering application 302 that helps a user gather thecontent on their personal computer with minimal effort; (2) a centralserver 304 (which may include many servers) having a web repository thatstores and organizes friends, shared links, and other similarinformation, and; (3) a television client 308 that presents a navigationmodel, plays the content, and allows sharing from a television.

The Computer Client

Currently there are at least three common ways that the user discoversnew video content when they are away from the television: they see it ata website such as YOUTUBE, they see embedded video content onthird-party sites, or they receive a link via email. By providing onetool where a user can quickly gather links from various disparatelocations, the system 300 simplifies this process. A client tool, suchas the gathering application 302, can give users the ability to sharewithin the system 300 itself. For example, if someone has e-mailed auser links (outside the system) or sent a user links from a gatheringapplication on another computer, the user can see the links and sharethem with others in the system 300 social network simply. Such a systemmeets the social need of sharing as easy as fulfilling your personalneed of collecting content.

In its simplest form, the gathering application 302 consists of a shortlist of a user's friends. People are represented as a unique id, theiremail, their name, and an icon. The system 300 creates a friend list andallows a user to present them alphabetically or in frequency of access.The first element in the list is a user's icon, making it easy toquickly gather content for later viewing and sharing. Adding a friendsends the person an e-mail and if they accept, then the friendship isconsidered mutual. Dragging a video link from the browser and droppingit on a friend sends the link to the internet repository indicating whenthe link was shared, by whom, and to whom. Clicking on the face of theselected user brings up further details about the user. A user can alsosearch over this list, for example, to locate all video with “surfing”in the title. This is a familiar format that can scale and supportscommon filtering, sorting, and annotating.

Central Server and Internet Repository

In the initial prototype system, the internet component in the centralserver 304 is basically used as a simple data storage that can beaccessed through standard internet protocols. The repository uses asimple relational database to hold the list of people, friends, groups,group members, and shared content. Several advantages arise from usingthis model. First is that it is a familiar and well implemented model.Second, the system 300 can use http to get and send data, and access itfrom a variety of places with a variety of devices. Finally, therepository allows the system 300 to do sophisticated storage, datamodeling, and complex analyses (e.g. clustering animations or cachingprominent content in multiple data centers/head ends), when the needarises.

The repository may store only simple content such as links, and can alsobe supplemented with a binary storage system that actually can storemedia files. The link storage may simply reference files that arepublicly accessible elsewhere on the internet, while a related binarystorage system may hold files that have been uploaded form clientdevices because they were not available elsewhere. The access to suchuploaded files may vary based on the preferences of the user whouploaded them—such as by being limited to access only by friends withwhich the user chose to share the media, up to full public access of themedia.

Television Client

The television client 308 can run on off-the-shelf hardware making iteasy to obtain, build, experiment with, and rapidly extend. In prototypeform, it may use an ordinary laptop, a remote control with drivers, andSVGA cables to ensure it works at standard “TV” resolutions. Thetelevision client 308 may obtain content and communicate to obtain andprovide recommendations over a network such as the internet 310.

The stack for television client 308 can include: a set-top box hardwareprototype built from off-the-shelf components; transport through theinternet; low level drivers for remotes, video & file systems; middlelevel file and “window” management that handles full screen video,overlays, and events; high level applications such as background contentfetchers, players, and watchdogs; a higher-level navigation model basedon zoomable shelves, trick-play, persistent searches, and televisionwidgets.

The client 308 can use an off-the-shelf remote control used to controlPC-based PVRs. FIG. 5 shows how a few buttons can be used to navigate inthree dimensions. Since zooming all the way in actually plays the video,the client 308 is able to overload the “select” button with “zoom in”and “back/undo” with “zoom out.”

Referring now to FIG. 3B, there is shown a schematic diagram of a system320 for sharing audio-visual program content. This figure focuses moreon the structure of an example content manager server 322, which maycoordinate the sharing of audio-visual content among multiple users. Thesystem centers around the content manager server 322, which connectsthrough a network 325 such as the internet, to a desktop computer 324, aflat panel television 326, and a mobile device 328 such as a smartphone.The server 322 may host s repository of shared media such as therepository discussed in relation to FIG. 3A and may be made up of one ormore different servers, but is shown here as a single server 322 forclarity.

The desktop computer 324 represents a computing device through which auser may surf the web and provide media program recommendations fortheir own television or for other users. The television 326 represents adevice on which a user may see recommendations from themselves or fromothers, may review audio-visual content, and may make simple actions toforward recommendations to other users, in manners like those discussedabove. The mobile device may provide both sorts of interaction—forexample, a user may surf the web or answer emails while commuting ontheir bus so as to identify links to media programs, and may then watchthe programs as soon as they get home. Likewise, they may watch programssuggested by others on their mobile device, depending on theirparticular mood.

Content manager server 322 is shown as including a number of componentsneeded to organize social sharing of media programs. A social mediafront-end 342 manages user interactions with the content manager server322. For example, the social media front-end 342 may provide code toclient devices for generating an interface like the gathering interface202 in FIG. 2 (which may be executed on the server 322 or on the clientdevices). The social media front-end 342 may also provide data toclients for populating such an interface. In addition, the social mediafront-end 342 may receive input from clients, such as indications that aparticular piece of media is to be shared or recommended to a number offriends of a user, or groups associated with a user.

The social media front-end 342 may supply such input information to auser data database 338. The database 338 may, for example, includeprofile information about particular users within a system, along withinformation about programs that they have recommended and that have beenrecommended to them. A related database—the relationship data database336—may include data reflecting links between users in the system 320.The relationship data database 336, for example, may include data likethat is normally provided in a social networking system to showparticular acquaintance relationships between users of the system (e.g.,friends, and friends of friends). The relationship data database 336 maybe used by the social media front-end 342 to provide clients with listsof friends and other acquaintances of a user associated with theclients. The relationship database 336 may also be accessed from anexternal sub-system such as from a separate social networking system.

An audio-video server interface 334 may be responsible for communicatingwith client devices that are to exhibit programs for the system 320. Forexample, the audio-video server interface 334 may, in response tocommands from the social media front-end 342 regarding shows recommendedby a user, or shows to be loaded on a user's television set-top box,schedule one or more downloads of content to be provided to users. Inaddition, the audio-video server interface 334 may respond directly torequests from a television, such as when a user of the televisionselects a program that has been recommended by a friend but has not beenpreviously downloaded to the television, to cause the content to beserved to the appropriate client.

The audio-visual content for programs may be fetched from audio-visualcontent database 340, which is shown here as a single database connectedthrough the server 322. The files in such a database may be referencedby URLs or other such pointers, so that they can be accessedconveniently. Alternatively, the audio-visual content may be stored awayfrom the server 322, such as at separate web sites (e.g., YOUTUBE) wherethe content resides natively. Also, certain content may be referenced inits native state at its original location while other content may beduplicated for use with the system 322, and stored on the system 322.The decision of which content to reference externally may depend, forinstance, on an analysis of the permanence of pages at the web site,where relatively stable web sites will be selected to be externallyreferenced because they are much less likely to delete a file so thatlater users would not be able to get to it.

A program scheduler 330 may further be used to integrate recommendationsprovided by the user himself, by friends of the user, or by automaticrecommendations such as by persistent searches, to produce one or moreprogramming schedules. A programming schedule may include a serialordering of programs to be displayed to a user, and may also inappropriate circumstances include advertisements to be played duringbumper segments between programs. The program scheduler 330 may, forinstance, sort a number of programs that have previously been identifiedby a user, so that programs may be displayed in an order most pleasingto the user. In one example, the programs may be ordered from highestrated to lowest rated programs (such as by star ratings assigned byviewers). In another example, the programs may be sorted from mostrelevant to least relevant, where relevance is determined by comparingconcepts associated with the programs to concepts associated with theuser. For example, a particular program may include metadata thatassociates the program with a particular demographic group, such asretired females. Such a program may be elevated in a queue for a userwho is a retired female. Alternatively, the programs may be ordered bytype, so as to create a consistent flow of programs throughout a viewingsession. As one example, serious programs may be grouped together, andcomedy programs may also be grouped together, with a separation betweenthe two types of programming. Other similar mechanisms for orderingprograms in a schedule may also be used. Users may choose to opt inbefore providing any such demographic data.

Information from the program scheduler 330 may be provided to an EPGgenerator 332. The EPG generator 332 may use such programming schedulesto produce an electronic program guide, that includes programming duringa certain period for a number of channels associated with a television.In this example, a schedule generated by the program scheduler 330 maybe provided to the EPG generator 332 for incorporation as one of thechannels in an electronic program guide to be provided to a user. Suchchannels may include broadcast channels like those in a traditional EPG,and may also include custom channels arranged by or for a user or users.For example, if a user trusts the tastes of another user, they cansimply let their television run all evening on a channel created by thatother user—with advertising revenue shared by the programmer, thecreators of the programs that are shown, and the operator of the system320.

Portable program module engine 337 may be provided in content managerserver 322 to generate program modules such as gadgets or widgets, to bedisplayed on a user's television. The portable program module engine 337may serve code for generating displays over the video on a television,such as by using a borderless Web browser defining certain IFramed areasfor the display of the content, where the displays may be generated soas to appear to be overlaid on the video. In this manner, the portableprogram module engine 337 may operate similar to systems such as theiGoogle homepage that permits users to select gadgets that they wouldlike to have displayed to them on a standard page.

One such gadget is shown in FIG. 7 overlaid on a playing video 700 in asemi-transparent manner. A particular module 702 includes JavaScript andHTML code for generating a current weather indicator. In this example,when a client device determines that the module is to be loaded anddisplayed, it may gather the code for the module and provide it to theweb browser in an element like an HTML IFrame, and may run the codeinside the IFrame. The code, when run, may obtain information, such asthe local temperature currently in San Francisco, in this example.Particular up-to-date content for the module 142 may be obtained by codein the module calling a live source connected to the Internet.

A variety of other portable program modules may also be implemented. Forexample, a module may be fed information about breaking news and may popup over a display of a program when important news occurs—providing aneffect like that provided by broadcast networks, but over the top ofstored, on-demand media content. Other program modules that have notbeen written to operate with a television-based interface may also beconverted, including on the fly, as described more fully below.

Other portable program modules may obtain information from the clientdevice to supply a user with targeted information. For example, aportable program module may obtain information identifying the programthat is currently being displayed by the system, and may makerecommendations for other similar programs (e.g., “If you like X, you'lllove Y”). Such information may also be used to identify other users thatare currently watching the same program so that, for example, the usersmay be entered to a common chat room where they can discuss the programwhile it runs and afterward. Another portable program module may obtaininformation about products shown in a program and may search forinformation about similar products that the user can purchase. Forexample, a video may be coded with areas on the display where productsare shown, and a viewer of the program may move a cursor around thedisplay to highlight a product and have a portable program module pop upand list a number of stores where the product can be purchased. In asimilar manner, an identifier for a product, or keywords associated witha program may be obtained by a portable program module that passes theidentity of the program to a central server, and the portable programmodule can then pass such keywords to an actively changing web site suchas eBay to obtain information about current, ongoing auctions for theproduct or similar products, and to thereby provide the user withup-to-date information.

As noted, the portable program module engine 337 may also be programmedto include auxiliary capabilities for reformatting program modules thatare not suitable for display on a television. For example, modules withsmall text and extensive content generally are not suitable fortelevision display because of the distance between a viewer and thetelevision. In addition, many computer displays have a different aspectratio than a typical television display, which may affect the manner inwhich a program module looks when displayed to a user. Also, televisionsare often of a different resolution than are computer monitors.

To adjust for such problems, and to permit portable program modules tobe migrated to television-based displays without the need for extensivere-programming of such modules, the portable program module engine 337may automatically convert the code for program modules so as to work ontelevision displays. An example process for such conversion is describedwith respect to FIG. 8A below.

In this example, the televisions widgets use GOOGLE GADGETS, an openarchitecture that includes, for the most part, HTML and JavaScript. Thismeans that televisions widgets can be derived from modules that workunmodified on tens of thousands of existing websites, including theGOOGLE homepage, desktop products like GOOGLE DESKTOP, VISTA sidebar andMac OSX Dashboard, and enterprise portal products from companies likeIBM. Going the other way, thousands of GOOGLE GADGETS written for theseother types of products will work on television sets, where appropriate.

Users can then configure channels to hold not only video content butalso program modules, and may define whether they are present all of thetime or just on pause (users can also configure them to be presentacross all channels and may also include alert notification). In FIG. 7,for example, the weather data is being provided by an RSS feed. Forexample, a programmer of a personalized channel may himself identifyproducts in a program and provide for a portable program module thatpermits selection and purchase of the items, if a seller of the items iswilling to give the programmer a referral credit for viewers whopurchase the item. Programmers could also program other sorts ofwidgets, such as a snow condition reporter to be displayed during aWarren Miller skiing video.

A recommendation engine 343 may serve to receive recommendations fromusers to other users and to generate automatic recommendations. Thecombination of powerful search (e.g. fast indexing, handling stemming,crowding, etc.) and good recommendation engines help the user deal withscaling and content discovery. Recommendation engines have been aroundfor many years and a good taxonomy may be found at Montaner, M. et al.,“2003: A Taxonomy of Recommender Agents on the Internet,” in ArtificialIntelligence Review, 19, pp. 285-330. Arissono, L. et al., PersonalizedDigital Television, Kluwer Academic Publishers, The Netherlands (ISBN1-4020-2163-1) presents several articles around recommendations forElectronic Program Guides (EPGs).

The system 320 may generate a number of different outputs that can beused with devices 324-328. For example, a social interface 346 a likethe sharing application 202 of FIG. 2A may be provided to allow users toshare media content with other users.

FIG. 3C is a block diagram of a media display management system 350 formanaging content delivery to a television device, which may includevarious forms of video monitors equipped to deliver broadcastprogramming in various manners, such as over the air, via cable, orsatellite programming. System 350 is shown as involving a television 354and set top box 352, which are shown separately for clarity and becausethe two have traditionally been provided separately, but can also beintegrated with each other.

Set top box 352 accepts a number of inputs. For example, terrestrialover-the-air broadcasts, such as in digital form (at various standarddefinitions), may be received via antenna 361. Other information mayappear from paid broadcaster 360, which may include standard cabletelevision operators or satellite television. Additional media contentmay be received through network 356 such as the internet, from server358. Server 358 may represent various forms of servers, including aserver for managing user accounts, tracking relationships among user,and performing other administrative functions, such as content managerserver 322 in FIG. 3B. Server 358 may also include various media serversdelivering audiovisual content, such as in various standard streamingformats and by other approaches.

Fetcher 364 is controlled by other components in set top box 352 toacquire and store, or cache, audiovisual content from the varioussources to which set top box 352 is connected. Fetcher 364 may include,for example, media such as a hard disk drive or other storage forholding a sufficient amount of media content so as to permit convenientviewing of the content. Where system 350 is designed to provide contentvia streaming only, persistent storage in fetcher 364 may be reducedsubstantially or eliminated.

UI manager 366 may include a number of components for controlling themanner in which graphical objects are displayed on a television monitor.For example, an events manager make keep track of scheduled orunscheduled events with respect to media display on the system 350.Scheduled event may include, for example, scheduled downloads of contentsuch as by causing fetcher 364 to request content, and may also includeunscheduled events such as requests received from a user via a remotecontrol (via remote control manager 362, which may take a standardform). An overlays manager may also be provided, which may cause anumber of graphical elements to be added to a media display ontelevision 354, such as menus and other objects.

A shelf manager and a shelves manager may also be provided. A shelfmanager may control a user's interaction with programs arranged in aparticular shelf displayed to the user. For example, the shelf managermay show a number of media files in a linear array, with blanksrepresenting bumper content between each file. The shelf manager mayreceive commands from a user to move forward or backward between thevarious files, such as commands entered to direction arrows on atelevision remote control (which may include universal remotes). Ashelves manager may control actions across multiple shelves. Forinstance, a shelves manager may determine which shelves are shown andthe bases on which those shelves are organized. As one example, aseparate shelf may be shown for each friend of a user. If the user doesnot have many content items recommended by friends, the items from allthe friends may be merged onto a single shelf, and in somecircumstances, an identifier for each fiend, such as a photo of thefriend, may be shown with a fixed frame of the media content (see topshelf in FIG. 1B).

In addition, other shelves may be organized from other sources. Forexample, one shelve may contain the top rated or most viewed programs,while another shelf may contain programs recommended by the user himselfor herself. For example, a user may browse and select media files duringtheir commute, on a portable device, or over their lunch hour, on adesktop computer, and indicators for the files may be associated withtheir account so that the files are queued up for them when they gethome from work. The limited capabilities (e.g. relatively low bandwidthand screen resolution) of the mobile device may be acceptable forselecting still images, while higher performance activities such asdisplaying motion video may be reserved for more capable home equipment.In addition, users may indicate that they are part of a group, such as aclub or a group of subscribers, and media recommendations may be made byone or more persons or systems for the group. As one example, a userwith interests in aviation may join an aviation group, and may thus havea shelf that shows various spectacular aviation videos. The shelf may beorganized by an organization, such as an organization that is given theopportunity to include advertising on the shelf in exchange for theirorganizational work, or by a person, such as a volunteer (or someone whosells advertising time and makes themselves effectively a programmanager) with recognized skill at organizing entertaining programming.

In addition, a search result such as “dog videos” or “dog images” or“dog” can become “persistent” every day (or some time unit), so that newmedia files about dogs gets added to the shelf. A user can tag items onshelves and “lock” them so that they do not go away. Thus, if a usersees an interesting talk on a computer programming topic, they can saveit and build up a collection/shelf. So, a shelf can represent apersistent, on-going search and a locked down set of results, andfacilitate a hybrid of both.

A content manager 372 may accept requires from other modules to storeand to deliver content and may cause the content to be served in suchsituations. In general, the content manager 372 operates to communicatethe identity of the user's content to other components in the system350.

Browser 368 may provide standard browser functionality, and may providefor full screen video display in appropriate circumstance, and maygenerate graphical overlays for video. The browser may be configured,for example, to generate gadgets or widgets over particular areas of atelevision screen. The browser may read standard mark up code, such asHTML with JavaScript references, and may provided asynchronous operationwhile data requests are being served in the background via XML datatransfer.

A friend manager 370 keeps track of friends for a user or users 370. Thefriends may be stored as an array of other users with which a first userhas identified a relationship. The friends information may be imported,for example, from social networking sites like ORKUT or MYSPACE so thata user need not recreate their main list, and so that a user's variousfriends lists may be constantly coordinated. Each record in a friendsdatabase accessed by friend manager 1529 may include one or more fieldsreflecting a name of a friend, an identifier for the friend (e.g., a login ID or e-mail address), an image associated with the friend (e.g., aphoto of the friend or a picture otherwise selected by the friend to berepresentative of them), and any other information needed to receiverecommendations for media files from friends and to provide suchrecommendations to friends.

Portable program modules 355 a-c may also be shown over or next to arunning media program 355 on display 354. The modules, as mentionedabove, may carry a number of forms of content, including content that istargeted to the content of the media program 355. The selection of whichmodules to show may be made by a user, such as by setting up a sidebarof modules similar to the manner of establishing a GOOGLE DESKTOPlayout, and/or made be made in whole or in part by the program 355itself. Also, user-established modules may be replaced periodically andtemporarily by program-initiated modules.

In addition, computing structures may be provided to convert existingportable program modules from a format suitable for desktop computers toa format or formats suitable to televisions. In particular, changes inresolution between the two types of displays may require certain amountsof transcoding, as may the differences in typical input mechanisms—i.e.,mouse and keyboard vs. remote control.

The transcoding may be fully or partially automatic. In general, thetranscoding will involve altering of code, such as standard mark upcode, in the module that is responsible for display of information andinputting of information. For example, a first step in transcoding maybe to change font sizes for text displayed by a module and imagesrendered by the module. In addition, mark up code may be added to amodule so that four-direction inputs from the pressing of navigationarrows on a remote control may cause an input indicator (e.g., cursor)to move from one input control to another on the displayed module. Wherestyle sheets are provided to control the appearance of the displayedmodule, the style sheet may be modified appropriately also. In addition,certain changes may be made automatically, with the module displayed toa user in a mocked-up form, so that the user may see the effect of eachindividual change, and may approve or disapprove of each change, or mayaccess the code to change the change. In addition, changes to the codemay be made by a set top box when the code is acquired or at run time,or the display of a module may be affected, e.g., shrunk on a televisiondisplay, after it is generated from the code. A process for transforminga portable program modules in such a manner is shown, for example, withrespect to FIG. 8A below.

The code for the portable program modules may then be stored in aweb-accessible database for access by set top boxes. The boxes may beprogrammed to access a URL associated with a page that shows a selectionof available portable program modules, such as clock, weather, or othermodules. Such types of available modules may be seen by establishing aGOOGLE user account and a personalized home page, and selecting “addstuff.” When a user selects a particular module, the set top box maymake a request for the code associated with the module and may downloadthe code to the box.

The modules may incorporate JavaScript that makes requests for data fromvarious sources, using, for example, AJAX techniques. The requests mayreturn data, such as current information data, that may be returned tothe module to affect displays of the module, such as when layering thedisplayed module over operating video.

The displayed modules may be positioned according to a stored mechanismon the set top box. For example, default starting points (e.g., for alower left corner of a modules) for each module may be established ineach corner of the television screen, with the module appearing in adefault corner unless the markup code specifies a location. A user mayalso change the default location, such as in a manner like traditionalsystem that permit movement of a PIP window.

FIG. 4 shows a schematic diagram of a system 400 for sharing anddisplaying media content, where the particular components here mayrepresent the portions of the sub-systems shown in FIG. 3A. Asdiscussed, the system 400 generally includes a personal computer client402, an internet repository 404, and a television 406 running atelevision client 407—all communicating through a common API 416. Thepersonal computer client 402 permits users to assign content to theirown account or to their friends, among many other things. The internetrepository 404 keeps track of such content assignments, and ofrelationships between users (perhaps by requesting information from aseparate social networking sub-system). The television client 407fetches content from the internet repository 404 and accepts inputs froma user with respect to content the user would like to play and contentthe user would like to share (which involves uploading indicators forsuch selections to the repository 404).

Referring now more specifically to the personal computer client 402, theclient 402 generally includes a small number of elements run in astandard computer operating system. In this example, the elementsinclude s UI manager 408 and a friend manager 410. The friend manager410 takes a list of friends and related information about the friendsfrom the internet repository, and provides the information for displayin a variety of ways. For example, the friends may be sorted byrelevance, such as how often the user has shared content with theparticular friends. In a like manner, groups and devices may berepresented as friends in the system, as discussed with respect to FIG.2A above. The UI manager 408 may take information from the friendmanager 410 and format it for display in a convenient manner for reviewby a user and for addition of content by the user, such as in the formof gathering application 202 shown in FIG. 2A.

The UI manager may take a variety of forms and may control the manner inwhich programming is displayed to a user and the manner in which a userinteracts with the client 402 in order to share content with theirdevices and with other users, such as is shown in FIG. 2A.

Referring now to the internet repository 404 of FIG. 4, there again isshown a limited number of components for clarity. However, different andadditional components may also be provided. A SQL database 414 is shownas a simple database to store information about relationships betweenpairs of users and groups, and between users and particular content. Forexample, the database 414 can store links between users and also linksfrom users to audio-visual content, which may be stored with theinternet repository 404 or elsewhere on a network such as the internet.A group of JAVA servlets or other appropriate code may be provided onthe internet repository to interact with programs running on thepersonal computer client 402 and the television client 407, so as toprovide functionality like that discussed above and below. Variousimplementations of the code to interact with the clients may beemployed.

Referring now to the television 406 and associated television client407, there is provided three major applications: a fetcher 418 thatscans for links and retrieves the content as a background task; a playeras part of a web browser 420 that navigates and presents content; and awatchdog 419 that makes sure everything is running properly. Using thefetcher 418, the client periodically downloads links from the internetrepository 404 and resolves them. It compares the recommendation linkswith a local list to see if the content has already been downloaded. Ifnot, then in a model similar to digital video recorders (DVRs, e.g.,TIVO) it copies the video content to the local drive for later personalviewing. Where fetching is done in batch rather than at play time,longer playing, higher quality content may be feasible to download overa connection having restricted speeds. The fetcher 418 is also capableof sending logging information back to the internet repository 404; thisalso presents an opportunity to use various DRM models (play-once,expire-after, etc.).

The watchdog 419 uses a simple heartbeat model. Each applicationregisters how frequently it sends a message to the watchdog. If thewatchdog 419 does not receive a heartbeat consistently, it assumes theapplication is inoperable, kills any remnant processes, and restarts theapplication. The watchdog 419 also starts all of the processes on areboot and accepts commands to terminate all processes.

In this implementation, display of programming occurs through a standardweb browser that is displayed using a Flash player with a full-screenwindow having no borders (or an H.261 player, such as for mobileapplications), so as to fill the entire television screen with video.Such a display may also permit the overlay of interactive portableprogram modules (e.g., gadgets) on top of the playing video, such as byusing a JavaScript engine and HTML renderer. A UI manager 426 may betasked with organizing the overall visual display of the system. Inparticular, the UI manager 426 may manage the location of itemsdisplayed on each shelf in an interface an may also manage thetransition from showing bumpers or shelves to displaying an actualprogram. The UI manager 426 may also manage the manner and location fordisplay of portable program modules such as gadgets or widgets.

A remote control manager 428 having an API may interface with the UImanager 426 to translate restricted inputs that are received from aremote control. For example, selections around a 4-way interface may beinterpreted by the television client 407 as directional inputs to beapplied to a cursor on the UI display.

A friend manager 422 receives information about friends of a userdownloaded from the internet repository 404, either with or withoutassistance from a more general social networking system. The friendmanager 422 presents a list of friends and associated meta data to theUI manager 426 when requested. For example, the UI manager 426 mayinitially seek information on all of a user's friends, and may then seekdetailed information about a particular friend if the user selects thatfriend on the television screen (e.g., FIG. 1E).

A content manager 424 organizes the content downloaded to the televisionclient 407. For example, the content manager 424 may associateparticular pieces of content with particular users, so that, when the UImanager 426 seeks to display a shelf of programs that were recommendedby a particular friend, it can get images for such programs via thecontent manager. Further descriptions of components in an exampletelevision client are discussed above with respect to FIG. 3C.

On first examination, the problem of automatically recommending videosis similar to many other recommendation challenges on theweb—recommending products to buy based on previous purchases (such ascommonly done on Amazon.com), recommending movies to watch (i.e.Netflix.com), or even recommending people to date based on their statedprofiles (i.e. Match.com). With respect to the video recommendationtask, at a high-level, the approaches can be divided into broadcategories.

The first approach is to cluster the items based on item descriptions(i.e. based on the description of the video given by the user whouploaded the item). Then when a user watches a video, other videos inthat same cluster can be recommended.

The second approach is to match either explicit or derived descriptionsof the user to the keywords describing the videos. For example, if theuser often watches videos for which the description includes the words“music”, “rap”, etc., the system can infer a simple keyword-basedpreference-vector based on this, and recommend new videos that alsocontain overlapping terms.

The third approach is “content-agnostic” recommendations. Rather thanlooking at attributes about the videos, such as keywords, statisticsabout views of the videos are maintained. For example, a system can keepan anonymized (but unique) id associated with each user; the videos seenare then associated with each user. To provide a recommendation, severaloptions are available. After a video is seen, the system can look at“co-watch” statistics—for users that also saw video-A, what other videosdid they also watch? Additionally, the system can limit the set of userfrom which these statistics are gathered by examining users who areeither explicitly stated “friends”, or users who are determined to havesimilar interests (for example, based on approach #2).

A combination of approaches can also be used. Further, all of theapproaches can also be used in conjunction with ratings like thosegathered on YOUTUBE. It is only when the particular task is examined indetail that it becomes evident that the characteristics of this domainmake approach #3 particularly useful. First, the descriptions of thevideos are often terse and provide little to categorize videos (i.e.“summer vacation—twins”). Second, generating automatic descriptions ofvideos is currently beyond the scope of what automatic computervision/annotation research is capable of. Third, with neither keywordsnor higher-level derived semantic descriptions of videos, ascertainingvideo-content subject similarity is not currently a solvable task (i.e.,video-A and video-B are similar in subject matter). However, approach#3, which is con-tent-agnostic, does not rely on content description orsubject similarity; instead it relies on robust user-statistics. Thisfits well with YouTube: as the large number of users, views and ratings,allows us to obtain reliable watching statistics.

At a simplistic level, recommendation systems that are not based onsimple keyword matching for this domain can be created by recommending,for user-X, videos that are often watched by other users who have alsowatched videos that user-X has watched. However, in applying thisapproach, two problems may immediately arise. First, for popular videos,many copies of the video may be uploaded; there-fore, when recommendingvideos to watch, it is important to ensure that we not recommendmultiple copies of the same video. Further, it is important to calculate“co-watch” statistics on videos which account for duplicates (forexample, co-watch statistics should account for the fact that somevideos have hundreds of duplicates on-line). Unfortunately, simplematching schemes, which may be based on the assumption that exactmatches exist, will not work. Differences in compression, start/stoptimes, glitches, etc, make exact matches fail. Instead, we havedeveloped methods based on Wavelet-matching to capture inexact matches.Secondly, there will be a problem of recency; when a new video isuploaded, it will not have the same magnitude of co-watch statisticsthat older videos will; and more users will have seen older videos thana recent video, simply based on the longer time that it has beenavailable. To address this, techniques that age discount recommendationcandidates can be used.

In summary, currently recommendation systems that are based onstatistics of viewings, rather than in-depth content analysis can beapplied to video recommendations. Within the example system here, theymay appear to the user as being recommended as another friend, or withina new shelf. Nonetheless, as either more image/video content-analysistools are developed or more users annotate their videos to ensuresearchability of their content, the more other, more traditionalcontent-based recommendations tools will become increasingly effective.

FIG. 8A is a flow chart of a process 800 for converting a portableprogram module, such as a gadget, from a computer format to a televisionformat. Such a process may be executed automatically, orsemi-automatically, to provide for portable program modules that areappropriate for television display and interaction, from modules thatare not so appropriate. In this manner, the vast number of portableprogram modules, such as gadgets, that have been written and are stillbeing written for the desktop environment, may easily be used in thetelevision environment.

The process begins at box 802, where a desktop portable program modulesis initially identified. Such identification may occur via a variety ofmechanisms. For instance, a user wishing to have the portable programmodule displayed on their television may identify the portable programmodule at a site such as the IGOOGLE set up site. Alternatively, theportable program module may be identified by a browser operating with atelevision, where the browser attempts to call up the desktop version ofthe portable program module, and recognizes that it is not fullyappropriate for display on a television. In such a situation, theconversion may occur in real time and automatically.

At box 804, the process 800 parses the DOM for the desktop version ofthe portable program module. Such parsing may give the process 800 moredirect access to objects that define the portable program module. Theprocess 800 may then identify relevant formatting code for the page andtranslate or convert such code (box 806). For example, where theformatting demands a font of a particular size, the process 800 mayreduce the size of the font. Also, various objects may be removed formthe portable program module if they are determined by the process 800 tobe unnecessary. As one example, long textual explanations may beremoved, as may images that are determined not to provide additionalcontent to a portable program module (e.g., if they are not selectableor are repetitions of textual content). Such modifications may alsoinvolve modifying definitions in style sheets for content in a portableprogram module.

At box 808, the process 800 identifies relevant content and reformatsthe content. This reformatting can be distinguished from thereformatting of format, rather than content, in box 806. To reformatcontent, for example, the portable program module may be looped throughusing a number of heuristic rules for transformation of a portableprogram module. For example, a first loop may look for images in theportable program module, and may reduce the pixel size of such images byan amount proportional to the changed that is determined to be needed soas to make the portable program module as visible on a television as itwould be on a desktop monitor. Also, there the portable program moduleis to be converted in aspect ratio (e.g., from 4:3 for a monitor, to16:9 for a television), the figure may be cropped at its top and bottomor sides, or may have its own aspect ration changed by known mechanisms.A second loop may look for alphanumeric textual content, and maydetermined that certain strings are too long for display on atelevision. In such a situation, particular portions of the strings maybe excised, and only those portions used in a television-based portableprogram module (e.g., bolded portions or portions determined to bekeywords). Other content may be affected by similar editing rules, whichmay be implemented automatically

Each such automatic reformatting operation may be tested with a user.For example, each change may be stepped through in sequence anddisplayed to the user, and the user may confirm whether they would likethe change to be made or not. The final revised portable program modulemay then be displayed to the user, either on a television or as if itwere on a television, and the user may affirm it or make additionaledits to it.

At box 810, the revised portable program module has its code and itscontent repackaged. Such action may simply result in the generation ofrelevant HTML, CSS, and JavaScript files, in some examples, in additionto revised image files as noted above. Such files may also be savedtogether in an appropriate area so that it may be conveniently accessed,such as in a file directory structure that contains television-focusedportable program modules, but is parallel in format to a directorystructure for corresponding desktop-focused portable program modules.

At box 812, the code and content is provided for display on atelevision. Such an action may occur by the mechanisms discussed above,such as by a television-based client device making a request for theportable program module at the location in which it has been stored. Ifthe quality of the converted portable program module has not yet beenconfirmed, then the user of the television may confirm whether theportable program module looks appropriate on a television. At such apoint, the portable program module may be made to other users who areusing the televisions to view network-based media content as describedelsewhere herein.

FIG. 8B is a flow chart of a process 820 for sharing audio-visualcontent among friends and/or devices. The process 820 may be performed,for example, by systems such as those shown in FIGS. 2A-2C and 3A-3C.The process 820 starts at box 822, where a request is received form auser to display a friends list. The request may occur, for example, bythe user launching an application such as a chat application or thesharing application 202 of FIGS. 2A-2C. The process 820 may thenidentify devices and users (e.g., friends) that are associated with therequesting user. Such a determination may be made simply, such as byaccessing a profile for the requesting user that includes a list ofidentifiers for other users and/or devices. The determination may alsobe made in whole or in part by providing an identifier for therequesting user to a separate social networking system that may thenreturn identification information for the other friends. In one example,the friends information may be provided by a social networking system,and information about a user's other devices may be provided from theuser's profile.

The process 820 may then use the retrieved information to serve data fora friends list to an application associated with the user. Such data mayinclude a name of each friend, an identifier for each friend, a photo ofeach friend, and status information about the friend, such as whetherthe friend is currently on-line. The application may then use suchinformation to display a toolbar or similar representation, to which theuser may drag indicators for media content, such as URLs.

At box 828, the process 820 receives a selection from the userassociating a media program with a member or members of the list. Such aselection may occur, for example, by the user dragging from a link for amedia file on a web page to a user or users shown with the launchedapplication. The user may also drag a file from a file manager on theuser's operating system to the application in a familiar manner, and thefile may be uploaded using the application.

At box 830, the information for the friend and media item or program maybe verified. For instance, a communication may be made between theclient application and a central server that may organize media sharingamong friends. The central server may verify that the selected friend orfriends are truly friends of the user, and may also verify that theselected media is publicly available.

At box 832, the process 820 causes an indicator of the media program tobe sent to accounts associated with each selected friend. For example,with the friends identified using the application, the URL or otherinformation may be copied into a profile associated with each of thefriends, where the profile includes lists of friends of the friends,along with URLs that have been recommended by each of the friends. Ifthe media is not fully available to the public (e.g., if it is on thefirst friend's hard drive), it can be copied to a location in which itwill be publicly or partially publicly available, though restrictions onits use (each DRM) may be tracked and limits may be placed on copying inresponse to such restrictions. If the “friend” is actually a deviceassociated with the recommending user, the URL may be added to a list ofprograms for the device, and the device or a main system may initiatedownloads to the device, such as at a prescribed time each day.

In this manner, a first user may, in a simple manner, see media contentand share it easily with their friends. A central system may trackrelationships among users and recommendations made by first users forsecond users. The system may also, in appropriate circumstances, managethe delivery of the media to target devices. As a result, users mayreview content that has been programmed by the friends, who presumablyhave similar interests to them. In addition, if a group of friendsreviews similar content, such shared experience may further enhance therelationships between and among members of the group.

FIG. 8C is a flow chart of a process for sharing audio-visual contentamong friends. In general, while FIG. 8B was worded largely in terms ofactions that a server might undertake in a media sharing system, FIG. 8Cis expressed in terms of actions taken by a client. Of course, theparticular actions are provided for example only, and various actionsmay be performed by clients or servers, as is necessary or convenient.

The process 834 starts at box 836, where a client device receives a userrequest for the display of a friends list. The device may transmit thatrequest, along with an identifier for the user, to a media server thatis remote from the device (box 838), and may in turn receive informationon friends of the user and their related media (box 840). For example,the central server may access a profile for the user and may provideinformation about programs the user has recommended for his or herfriends, and/or programs the friends have recommended for the user. Suchinformation may then be displayed in a client application, such as inthe manner shown in FIG. 2C (box 842).

A user may then select to play media programs, and to play programs withinserted bumpers, at box 844. In such an example, the user may be actingas a media programmer who is organizing an entire evening ofprogramming, and is thus determining some or all of the advertising tobe shown with the programming. When the full programming of all theprograms and bumpers is organized by the user, such as when a fullevening of programming has been dragged onto a personalized channel andordered in a proper manner, the client may receive from the user aselection to share the media, and may in response transmit informationabout the programs to a media server (box 646). Other users that arefriends with which the first user wants to share the programming, orsubscribers to the user's programming service, may then have identifiersfor each of the programmers and bumpers inserted as a list in thefriend's user profiles. Alternatively, the programming may be storedonce at a central location, and pointers to the programming may be savedfor each friend or subscriber.

FIG. 8D is a swim lane diagram of a process for sharing audio-visualprograms among devices. The process here is similar to those justdescribed, but shows example steps that may occur on each main componentof a three-component system that includes a television client, acomputer client, and a one or more media servers, line those describedabove. Generally, the process shows steps that may be taken by a usersharing content, from their work computer or smartphone (e.g., duringtheir morning or evening commute), with their home television, and thenviewing the content on the television after they get home.

At box 850, a user at a computer client, such as a smartphone or adesktop computer, requests to be provided with a media connectioninterface, such as the application 202 of FIGS. 2A-2C. Such a requestmay launch a media sharing application, which may then obtaininformation for a media connection interface from the server (box 852),and then may use such information to display acquaintances and devicesassociated with the user (box 854). As the user surfs the web, they maysee a variety of interesting information, such as videos that they canwatch but do not have the time to watch (perhaps they are on lunchbreak) or the quality of device to watch (perhaps they are on a bus withtheir telephone). In such a situation, they may, for example, drag alink to the information to their sharing application or otherwise selectthe link (e.g., with a right click or other contextual menu selection)so that they can select other users, devices, or groups with which theywould like to share the link, and by extension, the information (box856).

The client may send information, such as identifiers for therecommending user and the users to whom they are making arecommendations, along with information for the link, among otherthings. The server may parse such a transmission to identify the medialocation form the URL, and the identities of the users with whom theprogram is to be shared (box 858). The server may then verify that therecipients are truly friends of the first user, and are also activeusers with the system, and may get the media, or at least check theaddress to ensure that the media is at the location (box 860). Theserver may then report the results of the check back to the client (box862) and the client may display to the initiating user that the sharingoperation was a success or that it encountered errors (box 864).

With the sharing registered, the server may schedule a delivery of thecontent to the target in certain circumstances (box 866), such as whenthe target is a device belonging to the initiating user. The schedulingmay depend, in the first instance, on a schedule for the user, which maydefine a permissible window during which the content may be delivered(e.g., before the user gets home from work, or before they wake in themorning, or even before a particular day if they go entire days withoutwatching television).

The system may also update EPG data for the user, at box 868. Inparticular, if the user, when selecting programs for showing on theirhome television, is stacking programs one after the other, the programsmay be added to a personalized channel on an EPG, with a typicalstarting time for the user and the lengths of each of the programs andrelated bumpers controlling the length and timing of the fullprogramming. The media may then be delivered, according to the schedule,from the server to the television client (box 870), and may be stored ona local storage media such at the client (box 872).

Later, when the user returns home, they may access the client to showshelves of programs for them or an EPG for their personalized channel.They may delete certain programs or rearrange them, and may also addprograms recommended by their friends, before starting their viewing ofthe programs. At box 874, the user express a desire to begin watchingthe programming, and the client begins to play the media. As the mediaplays, of advertisements or other bumper material has not previouslybeen selected and downloaded, the client may send identifyinginformation for the playing programs to the server, which may select anddeliver advertisements and other information for the user (box 876). Forexample, interesting facts about a program that have been generated by afan of the program may be popped up during the playing of the program,using, for example a portable program module such as a gadget.Alternatively, or in addition, advertisements may be selected accordingto key words that have been associated with the program. For example, ifthe program has a “cars” tag, then advertisements from auto supplycompanies or automobile manufacturers may be displayed in the bumpers.

The next portion of this document describe related work in this area.Work in the area of Interactive TV (ITV) has been going on for years.Much of the work has been on personalized televisions especially aroundpersonal Electronic Program Guides (pEPGs). From a hardware platform andnetworked media standpoint, most work has been done around PersonalVideo Recorders (PVRs or DVRs), media centers, and more generally settop boxes (STBs). For example, the MICROSOFT MEDIA CENTER runs on PCsand computers-turned-PVRs. This system also integrates with gameconsoles such as the XBOX 360, allowing the streaming and transfer offiles.

While these devices are arguably the hardware convergence of computersand televisions, they do not adequately address the convergence,collision, and collaboration of the user's different activities in thesedifferent environments. The operations to collect and share are hiddenand not part of the main activity flow, but are instead relegated to anumber of third-party applications in the backwaters of the system.

To evolve these systems, we look at activities models, and attempt tointegrate and improve them across the platforms. Underlying the modeldescribed here, which is a particular implementation described and shownhere, is a variant on the Collect-Relate-Create-Donate (CRCD) model. Thesystem makes the ability to collect and donate on different platformssimple and a first class participant.

To understand what people do and how they watch TV, we refer to Lee, B.and Lee, R., “How and Why People Watch TV: Implications for the Futureof Interactive Television,” J. of Advertising Research, 35(6) (1995) andLull, J., “Inside Family Viewing: Ethnographic Research on Television'sAudiences,” London: Routledge, pp. 49-61 (1990). Lull identified sixuses of television as extensions of the household:

-   -   Environment (background noise, companionship)    -   Regulate (punctuate time, regulate activities)    -   Communicate (illustrate experience, common ground, reduce        anxiety)    -   Affiliate (verbal contact, family relaxing, reduce domestic        conflict)    -   Social learning (model behavior, solve problems, make consumer        decisions)    -   Competence (re-enforce roles, validation, facilitate arguments,        family solidarity).

The ability to create a playlist (or shelf) allows you to customize ittowards social needs—for example, building a channel of educationalvideo for your children and watching them as a family. SeeChorianopolous, K., “The Digital Set-Top Box as a Virtual ChannelProvider,” in SIGCHI '03 (Ape. 5-10, Fort Lauderdale, Fla.), ACM/SIGCHI,2003, pp. 666-67 for earlier work in virtual channels. And the conceptof shelves as a metaphor for information access has been around perCard, S. et al., “The WebBook and the WebForager: an InformationWorkspace for the World-Wide Web,” Proceedings of the SIGCHI 96, p. 111(Apr. 13-18, 1996). Other work related to this includes Phiz (Rowson, J.et al., “Discovering TVs Long Tail Through a Channel-Centric Model,”EurolTV 2005, Mar. 30-Apr. 1, 2005 (also HP Technical Report:HPL-2005-84) and Joost (http://www.joost.com). Phiz is an example ofcreating custom TV that retains the traditional channel-based model.Joost is an example of having the TV model on your PC. Finally, the XBox360 is a game-centric platform that has expanded to include watching HQvideo content (full length movies) and collaborating with fellow gamers.

FIG. 9 shows an example of a generic computer device 900 and a genericmobile computer device 950, which may be used with the techniquesdescribed here. Computing device 900 is intended to represent variousforms of digital computers, such as laptops, desktops, workstations,personal digital assistants, servers, blade servers, mainframes, andother appropriate computers. Computing device 950 is intended torepresent various forms of mobile devices, such as personal digitalassistants, cellular telephones, smartphones, and other similarcomputing devices. The components shown here, their connections andrelationships, and their functions, are meant to be exemplary only, andare not meant to limit implementations of the inventions describedand/or claimed in this document.

Computing device 900 includes a processor 902, memory 904, a storagedevice 906, a high-speed interface 908 connecting to memory 904 andhigh-speed expansion ports 910, and a low speed interface 912 connectingto low speed bus 914 and storage device 906. Each of the components 902,904, 906, 908, 910, and 912, are interconnected using various busses,and may be mounted on a common motherboard or in other manners asappropriate. The processor 902 can process instructions for executionwithin the computing device 900, including instructions stored in thememory 904 or on the storage device 906 to display graphical informationfor a GUI on an external input/output device, such as display 916coupled to high speed interface 908. In other implementations, multipleprocessors and/or multiple buses may be used, as appropriate, along withmultiple memories and types of memory. Also, multiple computing devices900 may be connected, with each device providing portions of thenecessary operations (e.g., as a server bank, a group of blade servers,or a multi-processor system).

The memory 904 stores information within the computing device 900. Inone implementation, the memory 904 is a volatile memory unit or units.In another implementation, the memory 904 is a non-volatile memory unitor units. The memory 904 may also be another form of computer-readablemedium, such as a magnetic or optical disk.

The storage device 906 is capable of providing mass storage for thecomputing device 900. In one implementation, the storage device 906 maybe or contain a computer-readable medium, such as a floppy disk device,a hard disk device, an optical disk device, or a tape device, a flashmemory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. A computer program product can be tangibly embodied inan information carrier. The computer program product may also containinstructions that, when executed, perform one or more methods, such asthose described above. The information carrier is a computer- ormachine-readable medium, such as the memory 904, the storage device 906,memory on processor 902, or a propagated signal.

The high speed controller 908 manages bandwidth-intensive operations forthe computing device 900, while the low speed controller 912 manageslower bandwidth-intensive operations. Such allocation of functions isexemplary only. In one implementation, the high-speed controller 908 iscoupled to memory 904, display 916 (e.g., through a graphics processoror accelerator), and to high-speed expansion ports 910, which may acceptvarious expansion cards (not shown). In the implementation, low-speedcontroller 912 is coupled to storage device 906 and low-speed expansionport 914. The low-speed expansion port, which may include variouscommunication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet)may be coupled to one or more input/output devices, such as a keyboard,a pointing device, a scanner, or a networking device such as a switch orrouter, e.g., through a network adapter.

The computing device 900 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 920, or multiple times in a group of such servers. Itmay also be implemented as part of a rack server system 924. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 922.

Alternatively, components from computing device 900 may be combined withother components in a mobile device (not shown), such as device 950.Each of such devices may contain one or more of computing device 900,950, and an entire system may be made up of multiple computing devices900, 950 communicating with each other.

Computing device 950 includes a processor 952, memory 964, aninput/output device such as a display 954, a communication interface966, and a transceiver 968, among other components. The device 950 mayalso be provided with a storage device, such as a microdrive or otherdevice, to provide additional storage. Each of the components 950, 952,964, 954, 966, and 968, are interconnected using various buses, andseveral of the components may be mounted on a common motherboard or inother manners as appropriate.

The processor 952 can execute instructions within the computing device950, including instructions stored in the memory 964. The processor maybe implemented as a chipset of chips that include separate and multipleanalog and digital processors. The processor may provide, for example,for coordination of the other components of the device 950, such ascontrol of user interfaces, applications run by device 950, and wirelesscommunication by device 950.

Processor 952 may communicate with a user through control interface 958and display interface 956 coupled to a display 954. The display 954 maybe, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display)display or an OLED (Organic Light Emitting Diode) display, or otherappropriate display technology. The display interface 956 may compriseappropriate circuitry for driving the display 954 to present graphicaland other information to a user. The control interface 958 may receivecommands from a user and convert them for submission to the processor952. In addition, an external interface 962 may be provide incommunication with processor 952, so as to enable near areacommunication of device 950 with other devices. External interface 962may provide, for example, for wired communication in someimplementations, or for wireless communication in other implementations,and multiple interfaces may also be used.

The memory 964 stores information within the computing device 950. Thememory 964 can be implemented as one or more of a computer-readablemedium or media, a volatile memory unit or units, or a non-volatilememory unit or units. Expansion memory 974 may also be provided andconnected to device 950 through expansion interface 972, which mayinclude, for example, a SIMM (Single In Line Memory Module) cardinterface. Such expansion memory 974 may provide extra storage space fordevice 950, or may also store applications or other information fordevice 950. Specifically, expansion memory 974 may include instructionsto carry out or supplement the processes described above, and mayinclude secure information also. Thus, for example, expansion memory 974may be provide as a security module for device 950, and may beprogrammed with instructions that permit secure use of device 950. Inaddition, secure applications may be provided via the SIMM cards, alongwith additional information, such as placing identifying information onthe SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory,as discussed below. In one implementation, a computer program product istangibly embodied in an information carrier. The computer programproduct contains instructions that, when executed, perform one or moremethods, such as those described above. The information carrier is acomputer- or machine-readable medium, such as the memory 964, expansionmemory 974, memory on processor 952, or a propagated signal that may bereceived, for example, over transceiver 968 or external interface 962.

Device 950 may communicate wirelessly through communication interface966, which may include digital signal processing circuitry wherenecessary. Communication interface 966 may provide for communicationsunder various modes or protocols, such as GSM voice calls, SMS, EMS, orMMS messaging, CDMA, TDMA, PDC, WCDMA, CDMA2000, or GPRS, among others.Such communication may occur, for example, through radio-frequencytransceiver 968. In addition, short-range communication may occur, suchas using a Bluetooth, WiFi, or other such transceiver (not shown). Inaddition, GPS (Global Positioning System) receiver module 970 mayprovide additional navigation- and location-related wireless data todevice 950, which may be used as appropriate by applications running ondevice 950.

Device 950 may also communicate audibly using audio codec 960, which mayreceive spoken information from a user and convert it to usable digitalinformation. Audio codec 960 may likewise generate audible sound for auser, such as through a speaker, e.g., in a handset of device 950. Suchsound may include sound from voice telephone calls, may include recordedsound (e.g., voice messages, music files, etc.) and may also includesound generated by applications operating on device 950.

The computing device 950 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as acellular telephone 980. It may also be implemented as part of asmartphone 982, personal digital assistant, or other similar mobiledevice.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms “machine-readable medium”“computer-readable medium” refers to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term “machine-readable signal” refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (“LAN”), a wide area network (“WAN”), and theInternet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

A number of embodiments have been described. Nevertheless, it will beunderstood that various modifications may be made. For example, variousforms of the flows shown above may be used, with steps re-ordered,added, or removed. Also, although several applications of the systemsand methods have been described; it should be recognized that numerousother applications are contemplated. Moreover, although many of theembodiments have been described in relation to televisions, that termshould be understood to include various forms of mechanisms for movingvideo and other content. Accordingly, other embodiments are within thescope of the following claims.

1-21. (canceled)
 22. A computer-implemented method, comprising: playing,by a computing device, first media content on a display device;determining, by the computing device, that the first media content hasended playing; presenting, by the computing device and in response tothe computing device having determined that the first media content hasended playing, a preview of second media content; determining, by thecomputing device, that the display device has presented the preview ofthe second media content for a predetermined period of time withoutreceipt of a pause user input or a skip user input; and playing, by thecomputing device and in response to the computing device havingdetermined that the display device presented the preview of the secondmedia content for the predetermined period of time without receipt ofthe pause user input or the skip user input, the second media content onthe display device.
 23. The computer-implemented method of claim 22,wherein: the first media content is a first video; and the second mediacontent is a second video.
 24. The computer-implemented method of claim22, wherein: playing the first media content on the display deviceincludes the computing device accessing first streaming media over theinternet and providing the first streaming media for presentation by thedisplay device; and playing the second media content on the displaydevice includes the computing device accessing second streaming mediaover the internet and providing the second streaming media forpresentation by the display device.
 25. The computer-implemented methodof claim 22, wherein: the computing device is configured to receive theskip user input while the computing device is presenting the preview ofthe second media content, and in response to receipt of the skip userinput, the computing device is configured to skip the playing of thesecond media content.
 26. The computer-implemented method of claim 25,wherein, in response to the computing device receiving the skip userinput while the computing device is presenting the preview of the secondmedia content, the computing device is configured to skip the playing ofthe second media content and present a preview of third media content.27. The computer-implemented method of claim 22, wherein: the secondmedia content is a video; and the preview of the second media contentincludes a still image from the video.
 28. The computer-implementedmethod of claim 22, wherein: the preview of the second media contentincludes a title of the video.
 29. The computer-implemented method ofclaim 22, further comprising: presenting, by the computing device, alist of media programs that shows the first media program and the secondmedia program adjacent to each other in the list; and receiving, by thecomputing device, user input that selects the first media program forplaying, and in response causing the playing of the first media contenton the display device.
 30. A computing system, comprising: one or moreprocessors; and one or more computer-readable devices includinginstructions stored thereon that, when executed by the one or moreprocessors, cause performance of operations that include: playing, by acomputing device, first media content on a display device; determining,by the computing device, that the first media content has ended playing;presenting, by the computing device and in response to the computingdevice having determined that the first media content has ended playing,a preview of second media content; determining, by the computing device,that the display device has presented the preview of the second mediacontent for a predetermined period of time without receipt of a pauseuser input or a skip user input; and playing, by the computing deviceand in response to the computing device having determined that thedisplay device presented the preview of the second media content for thepredetermined period of time without receipt of the pause user input orthe skip user input, the second media content on the display device. 31.The computing system of claim 30, wherein: the first media content is afirst video; and the second media content is a second video.
 32. Thecomputing system of claim 30, wherein: playing the first media contenton the display device includes the computing device accessing firststreaming media over the internet and providing the first streamingmedia for presentation by the display device; and playing the secondmedia content on the display device includes the computing deviceaccessing second streaming media over the internet and providing thesecond streaming media for presentation by the display device.
 33. Thecomputing system of claim 30, wherein: the computing device isconfigured to receive the skip user input while the computing device ispresenting the preview of the second media content, and in response toreceipt of the skip user input, the computing device is configured toskip the playing of the second media content.
 34. The computing systemof claim 33, wherein, in response to the computing device receiving theskip user input while the computing device is presenting the preview ofthe second media content, the computing device is configured to skip theplaying of the second media content and present a preview of third mediacontent.
 35. The computing system of claim 30, wherein: the second mediacontent is a video; and the preview of the second media content includesa still image from the video.
 36. The computing system of claim 30,wherein: the preview of the second media content includes a title of thevideo.
 37. The computing system of claim 30, wherein the operationsfurther comprise: presenting, by the computing device, a list of mediaprograms that shows the first media program and the second media programadjacent to each other in the list; and receiving, by the computingdevice, user input that selects the first media program for playing, andin response causing the playing of the first media content on thedisplay device.